{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4.3 滑动平均模型\n",
    "TensorFlow提供了tf.train.ExponentialMovingAverage来实现**滑动平均模型，它会对每个变量维护一个影子变量（shadow variable），这个影子变量的初始值就是相应变量的初始值，而每次运行变量更新时，影子变量的值会更新为：**\n",
    "\n",
    "$$shadow\\_variable = decay * shadow\\_varibale + (1 - decay) * variable$$\n",
    "\n",
    "$variable$为待更新的变量，$decay$为衰减率，用于控制模型更新的速度，可以看到decay越大模型越趋于稳定，实际应用中会设置成非常接近1（如0.999或0.9999）。\n",
    "\n",
    "为了使模型在训练前期可以更新得更快，ExponentialMovingAverage还提供了$num\\_updates$参数来动态设置$decay$得大小。如果在ExponentialMovingAverage初始化时提供了这个参数，那么每次得衰减率会是：\n",
    "\n",
    "$$min\\lbrace decay, \\frac{1 + num\\_updates}{10 + num\\_updates}\\rbrace$$\n",
    "\n",
    "\n",
    "下面给出ExponentialMovingAverage的一个简单样例（第5章中将会给出在真实应用中使用滑动平均的样例）：\n",
    "\n",
    "**1.定义变量及滑动平均类**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "v1 = tf.Variable(0, dtype=tf.float32)   # 注意所有需要计算滑动平均得变量必须是实数型\n",
    "step = tf.Variable(0, trainable=False)  # 模拟神经网络中迭代轮数，可以动态控制衰减率\n",
    "\n",
    "ema = tf.train.ExponentialMovingAverage(0.99, step)  # 初始化时定义了衰减率和step\n",
    "maintain_averages_op = ema.apply([v1])  # 定义一个更新变量滑动平均的操作，这里需要给定一个列表，每次执行时列表中得变量都会被更新"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. 查看不同迭代中变量取值的变化。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.0, 0.0]\n",
      "[5.0, 4.5]\n",
      "[10.0, 4.555]\n",
      "[10.0, 4.60945]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    # 初始化\n",
    "    init_op = tf.global_variables_initializer()\n",
    "    sess.run(init_op)\n",
    "    print(sess.run([v1, ema.average(v1)]))\n",
    "    \n",
    "    # 更新变量v1的取值到5\n",
    "    sess.run(tf.assign(v1, 5))\n",
    "    # 更新v1的滑动平均值。衰减率为 min{0.99, (1+step)/(10+step)=0.1} = 0.1\n",
    "    # 所以v1的滑动平均会被更新为 0.1 * 0 + 0.9 * 5 = 4.5\n",
    "    sess.run(maintain_averages_op)\n",
    "    print(sess.run([v1, ema.average(v1)]))\n",
    "    \n",
    "    # 更新step和v1的取值\n",
    "    sess.run(tf.assign(step, 10000))  \n",
    "    sess.run(tf.assign(v1, 10))\n",
    "    # 更新v1的滑动平均值。衰减率为 min{0.99, (1+step)/(10+step)=0.999} = 0.99\n",
    "    # 更新v1的滑动平均会被更新为 0.99 * 4.5 + 0.01 * 10 = 4.555\n",
    "    sess.run(maintain_averages_op)\n",
    "    print(sess.run([v1, ema.average(v1)]))       \n",
    "    \n",
    "    # 再次更新v1的滑动平均值，得到 0.99 * 4.555 + 0.01 * 10 = 4.60945\n",
    "    sess.run(maintain_averages_op)\n",
    "    print(sess.run([v1, ema.average(v1)]))   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
